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TRACKING OF A SUB IMAGE IN A SEQUENCE OF IMAGES 



FIELD OF THE INVENTION 

The present invention relates to a method of identification, in a succession of 
acquired images each formed from a matrix of pixels to a first format, of a following sub- 
image extracted from a following acquired image corresponding to a prior sub-image 
extracted from a prior acquired image, said sub-images being formed from a matrix of pixels 
to a second format smaller than the first format. 

BACKGROUND OF THE INVENTION 

Many video sequence acquisition devices are equipped with image 
stabilization means enabling the recorded image not to be disturbed by unwanted and 
unintentional movements of the operator handling the camera. 

In order to stabilize the image, some devices are provided with correction 
mechanisms acting directly on the optical elements of the camera. These devices are 
complex, bulky and not well suited to small items of equipment. 

There are also known video signal processing methods making it possible, by 
analysis of successive images, to stabilize the images recorded. 

Such methods provide for the camera to acquire digital images whose format 
is larger than the format of the images which are to be recorded, the latter then constituting 
sub-images of the main images acquired by the camera. 

In order to stabilize the image, it is known how to determine, from a prior sub- 
image taken from a prior acquired image, the following sub-image corresponding to the prior 
sub-image in the following acquired image. From the knowledge of the prior and following 
sub-images, it is possible to determine any movement that took place between these two 
corresponding images and to correct such movement in order to eliminate the effect of the 
unintentional movements of the operator. 

An image stabilization method is described in particular in the document 
EP-1 . 1 17.25 1 . This method makes provision for dividing an acquired image into a set of 
blocks covering the image and for determining, for each block, a movement vector 
characterizing the movement of this block from one image to the following image. A 
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movement histogram is created for all the blocks. After filtering, the results of the histogram 
are averaged and the movement vector is taken as being equal to the average of the filtered 
movement vectors of the various blocks. 

This method consumes a large amount of calculation time and therefore power 
5 since it is necessary, for each block in question, to determine a corresponding block in the 
following image. In addition, no algorithm is supplied in order to identify the following 
block from a prior block. 

It is an object of the invention to propose a method of identifying a following 
sub-image corresponding to a prior sub-image of a succession of acquired images able to be 
10 used in a method of determining the movement of an image which can itself be used in an 
image stabilization method, so that the image stabilization method requires only a small 
number of calculation operations and consumes little power. 

SUMMARY OF THE INVENTION 
15 To this end, the object of the invention is an identification method comprising 

the steps consisting of: 

- calculating, for the prior sub-image, at least one distribution of a 
characteristic quantity of each pixel for blocks forming a predefined partitioning of the sub- 
image; 

20 - calculating the same distribution for at least two would-be sub-images to the 

second format extracted from the following acquired image; and 

- determining the corresponding following sub-image from among the would- 
be sub-images, as the sub-image where the or each calculated distribution has the highest 
correlation with the same calculated distribution for the prior sub-image according to a 

25 predefined correlation law. 

Calculating a distribution in order to determine the corresponding following 
sub-image is simple and inexpensive in calculation power and nevertheless makes it possible 
to determine the corresponding following sub-image with satisfactory precision. 

According to particular embodiments, the method comprises one or more of 
30 the following characteristics: 

- it comprises the steps of: 

• calculating an extended distribution for an extended range of the following 
acquired image; 
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• calculating the correlations between the calculated distribution for the prior 
sub-image and a corresponding portion of the extended distribution for several shifts of the 
prior sub-image with respect to the following acquired image; and 

• determining the corresponding following sub-image as the sub-image of the 
5 following acquired image corresponding to the shift of the prior sub-image with respect to the 

following acquired image for which the calculated correlation between the distributions is the 
highest; 

- the blocks forming a predefined partitioning of the sub-image for calculating 
at least one distribution are rows and/or columns of the sub-image; 

10 - said characteristic quantity of each pixel is a parameter chosen from the 

group consisting of luminance, blue chrominance, red chrominance, red component, green 
component and blue component; and 

- the correlation law is defined as the inverse of the Euclidean distance 
separating two distributions. 

15 The invention also relates to a method of determining the movement, in a 

succession of acquired images each formed from a matrix of pixels to a first format, of a 
following sub-image extracted from a following acquired image with respect to a 
corresponding prior sub-image extracted from a prior acquired image, said sub-images being 
formed from a matrix of pixels to a second format smaller than the first format, the method 

20 comprising the steps consisting of: 

- identifying, in the following acquired image, the following sub-image 
corresponding to the prior sub-image by the use of a method as defined; and 

- calculating any movement between the prior and following sub-images from 
the position of the prior and following sub-images in the prior and following acquired 

25 images. 

It also has as its object a method of stabilizing images in a succession of 
acquired images each formed from a matrix of pixels to a first format, comprising the steps 
consisting of: 

- determining any movement in the succession of acquired images of a 
30 following sub-image issuing from a following acquired image with respect to a 

corresponding prior sub-image issuing from a prior acquired image, by the use of a 
movement method as defined above; 

- correcting said determined movement in order to take account of the effect of 
an intentional movement and to eliminate the effect of an unintentional movement; and 
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- adopting as the following image a sub-image of the following acquired 
image shifted from the prior sub-image by said corrected movement. 

It also has as its object a computer program product for a data processing unit, 
comprising a set of instructions for executing the steps of the method as defined above, when 
5 said program is executed by a data processing unit. 

Finally, the invention has as its object: 

- a device for identification, in a succession of acquired images each formed 
from a matrix of pixels to a first format, of a following sub-image extracted from a following 
acquired image corresponding to a prior sub-image extracted from a prior acquired image, 

10 said sub-images being formed from a matrix of pixels to a second format smaller than the 
first format, the device comprising: 

• means for calculating, for the prior sub-image, at least one distribution of a 
characteristic quantity of each pixel for blocks forming a predefined partitioning of the sub- 
image; 

15 • means for calculating the same distribution for at least two would-be sub- 

images to the second format extracted from the following acquired image; and 

• means for determining the corresponding following sub-image from among 
the would-be sub-images, as the sub-image where the or each calculated distribution has the 
highest correlation with the same distribution calculated for the prior sub-image according to 

20 a predefined correlation law; 

- a device for determining the movement, in a succession of acquired images 
each formed from a matrix of pixels to a first format, of a following sub-image extracted 
from a following acquired image with respect to a corresponding prior sub-image extracted 
from a prior acquired image, said sub-images being formed from a matrix of pixels to a 

25 second format smaller than the first format, the device comprising: 

• an identification device as above for identifying in the following acquired 
image the following sub-image corresponding to the prior sub-image; and 

• means for calculating the movement between the prior and following sub- 
images from the position of the prior and following sub-images in the prior and following 

30 acquired images; 

- a device for stabilizing images in a succession of acquired images each 
formed from a matrix of pixels to a first format, comprising: 

• a device for determining the movement as defined above for determining 
the movement in the succession of acquired images of a following sub-image issuing from a 
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following acquired image with respect to a corresponding prior sub-image issuing from a 
prior acquired image; 

• means for correcting said determined movement for taking account of the 
effect of an intentional movement and eliminating the effect of an unintentional movement; 

5 and 

• means for adopting, as the following sub-image, a sub-image of the 
following acquired image shifted from the prior sub-image by said corrected movement. 

BRIEF DESCRIPTION OF THE DRAWINGS 
10 The invention will be further described with reference to examples of 

embodiments shown in the drawings to which, however, the invention is not restricted. 

- Fig. 1 is a schematic view of video compression equipment using an image 
stabilization device according to the invention; 

- Fig. 2 is a flow chart explaining the image stabilization method according to 

15 the invention; 

- Fig. 3 is a schematic view of a prior image illustrating the calculation of the 
horizontal and vertical histograms used in the method according to the invention; 

- Fig. 4 is a schematic view of the following image of the image of Fig. 3 
illustrating the calculation of the horizontal and vertical histograms used in the method 

20 according to the invention; 

- Fig. 5 is a curve illustrating the method of correlating the histograms 
obtained during the implementation of the method according to the invention; 

- Fig. 6 is a curve illustrating the determination of the movement vector by 
implementation of the method according to the invention; and 

25 - Fig. 7 is a schematic view of a portable image recording and display device. 



DESCRIPTION OF PREFERRED EMBODIMENTS 

The image stabilization method according to the invention is particularly 
30 adapted to be used in portable equipment for acquiring video sequences, such as a mobile 
telephone equipped with an integral camera and means of transmitting the acquired video 
sequence after the latter has been compressed. 
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Fig. 1 depicts schematically the structure of video compression equipment 
used in a portable telephone, this equipment comprising an image stabilization stage 
according to the invention. 

The processing chain of the video compression equipment comprises digital 
5 image acquisition means 12 consisting for example of the lens of a camera associated with a 
matrix of sensors. The digital image acquired is formed from a matrix of pixels. Each pixel 
constitutes an image element characterized by various variables such as luminance, blue 
chrominance, red chrominance, red component, green component or blue component. 

The acquired images have a first format. They are for example 700 x 500 

10 pixels. 

The images which are to be encoded and compressed by the equipment have a 
second format smaller than the format of the acquired images. The images to be encoded 
therefore consist of sub-images issuing from the acquired images. The second format is for 
example 640 x 480 pixels. 
15 The acquired images of 700 x 500 pixels supplied by the acquisition means 12 

are processed by image stabilization means 14 according to the invention. These means will 
be described in detail later in the description. The images obtained at the output of the 
stabilization means are stabilized sub-images to the second format of 640 x 480 pixels. 

As is known per se, the succession of sub-images thus obtained is then 
20 compressed in the mobile telephone, before being transmitted. 

To this end, the sub-images are broken down by means 16 of breaking down 
into macroblocks of 8 x 8 pixels. These macroblocks are each sent to means 18 of estimating 
a movement vector of the macroblock with respect to its position in the previous sub-image. 
To this end, the compression equipment comprises means 20 of storing the previous sub- 
25 images. It also comprises means 22 of coding the differences between each macroblock to be 
coded and its antecedent in the previous sub-image. 

These means 22 receive as an input the movement vector of each macroblock 
produced by the estimation means 18, the corresponding macroblock issuing from the 
decomposition means 16 and the prior macroblock of the previous sub-image stored in the 
30 storage means 20. 

The information for each macroblock consisting of the movement vector and 
the coded differences are transmitted by the telephone to a receiver, where the information is 
decoded and the image sequence is reconstituted. 
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Fig. 2 depicts a simplified flow diagram of the algorithm used for the image 
stabilization means 14 according to the invention. 

Advantageously, the image stabilization means 14 are produced in the form of 
an adapted hard-wired electronic circuit, such as an ASIC. 
5 In a variant, the various stabilization steps are performed by a processor such 

as a DSP using an adapted program. 

According to the method according to the invention, the acquired images are 
processed successively as they are acquired. Thus a sub-image corresponding to a prior sub- 
image is determined for each new acquired image. 
10 The steps of the method illustrated in Fig. 2 are therefore implemented for 

each new acquired image. 

It is thus assumed that the acquired prior image denoted A t is known as well as 
the stabilized prior sub-image denoted S A t extracted from the prior acquired image A t . 

When the method is implemented, the following image denoted Ah-i is 
15 acquired at step 28. 

The image stabilization means first of all provide, at step 30, a calculation of a 
vertical and horizontal histogram of a characteristic quantity of the pixels of the following 
image A t +i issuing from the image acquisition means 12. 

The characteristic quantity considered in the histograms for each pixel is for 
20 example the luminance of each pixel, or the blue chrominance, the red chrominance, the red 
component, the green component or the blue component of each pixel. 

The histogram is established for predefined blocks of the following acquired 
image A*h. This acquired image has, as indicated previously, a format of 700 x 500 pixels. 

More rigorously, a distribution of a characteristic quantity of each pixel for all 
25 the blocks defined is established. Although the histogram is the graphical representation of 
this distribution, the term histogram will be used in the remainder of the description to 
designate this distribution, this term being more normally used in the art in question. 

For example, the horizontal histogram denoted H* +1 is composed of sums 
denoted Hi of the characteristic quantities of each pixel, for each of the lines 1 of the image 
30 At+i. In other words, each point of the histogram is defined for a line 1 by the following 
formula: 
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where: 

hi is the sum of the characteristic quantities of the pixels of line 1; 1 is the 



number of line 1; here 1 e [1; 500]; 

q 4 is the characteristic quantity of the pixel i in line 1, and NCI is the number of 
columns in the image; here NCI = 700. 

Likewise, a vertical histogram denoted is established for the image At+i. 
Each point H c of the histogram corresponding to a column c is defined by: 



where: 

H c is the sum of the characteristic quantities of the pixels in column c; 
c is the number of column c; here cg[1; 700]; 
qj is the characteristic quantity of the pixel j of column c, and 
NLI is the number of lines in the image. 

These histograms H* +1 and H* +1 are calculated for the entire extent of the 



images collected. 

An example of such histograms is depicted in Fig. 3, where the characteristic 
quantity of each pixel is the light intensity of the pixel. 

The image acquired to the format of 700 x 500 pixels is temporarily stored in a 
video memory of the FIFO (first in first out) type at step 32. 

The histograms of the previous extracted sub-image SA t of the sequence of the 
images which is to be stabilized are stored so as to be used subsequently at step 34. 

The horizontal SH* and vertical SH* histograms of the sub-image stored in 
the means 32 are obtained by extracting some of the larger histograms of the acquired images 
calculated by the means 30, as will be disclosed later in the description. It is therefore a case 
in fact of sub-histograms whose extent is limited to the sub-image SA t of the second format, 
as disclosed in Fig. 4. 

At step 36, a calculation of the distances between the histograms Ht +1 and H* +1 

and the histograms SHj and SH* of the following acquired image A t +i and of the prior sub- 
image SA t is made. To do this, the horizontal histogram and the vertical histogram of the 
acquired image and the vertical and horizontal histograms of the prior extracted sub-image 
are used. 



NLI 
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A similar processing is carried out for the horizontal and vertical histograms. 
Only the processing carried out on the horizontal histograms is described in detail below. 

A set of distances between the horizontal histogram SH* of the prior sub- 
image SAt and the horizontal histogram of the following acquired image A H i is 

5 calculated. These distances are calculated for would-be images extracted from the following 
acquired image At+i whose format is that of a sub-image, namely 640 x 480 points. The 
various would-be sub-images considered are offset from each other by one pixel over the 
entire extent of the prior acquired image, so that sixty successive would-be images are 
considered, as illustrated in Fig. 5, between the would-be sub-image and the prior acquired 
10 image A t . 

Each distance denoted Da for an offset of d pixels is defined by the difference 
between the sums of the characteristic quantities of the pixels of each block, here each 
column, over the extent of the histogram, namely by the formula: 



NCSI is the number of columns in the would-be sub-image; here NCSI = 640; 
y! +d is the quantity of the histogram corresponding to the column i + d for the 
following image At+i; 

yj sl is the quantity of the histogram corresponding to the column i for the 

20 would-be sub-image. 

The offset between the preceding sub-image SA t and the corresponding 
following sub-image SA t +i is determined, at step 38, from the results of the calculation of the 
distances. To this end, among the calculated distances the offset m corresponding to the 
smallest distance Da is adopted. 

25 The smallest distance is preferably determined from the curve formed by the 

distances Dd considering the point on this curve for which the second derivative is the 
greatest. A curve representing the second derivative of the function Da is illustrated in Fig. 6. 
In this example, the offset is equal to 28 pixels. 



30 acquired image At+i, the sub-image corresponding best to the prior sub-image SA t and in 
particular to determine the horizontal offset between these two images. 




15 



where 



The offset m thus determined makes it possible to determine, in the following 
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Steps 36 and 38 are implemented also for a vertical offset, considering the 

vertical histograms. 

The offset thus determined between the corresponding sub-images S A t and 
SAt+i is then corrected at step 40. This offset is corrected so as not to take account of any 
5 unintentional vibrations. A correction algorithm of this type is known per se and will not be 
described in detail. 

The corrected following sub-image S At+i is then extracted from the following 
image stored at step 42 from the corrected offset established at step 40. This following 
corresponding sub-image is then sent to the breakdown module 16. 
10 It will be understood that, with such a stabilization means, the calculations 

necessary for determining the offset between the two successive corresponding images is 
relatively small, which makes it possible for these calculations to be carried out with limited 
resources, in particular limited power consumption. 

Fig. 7 depicts another device implementing an image stabilizer by histogram 
15 according to the invention. This device is a camcorder comprising a set of sensors 112 

allowing the acquisition of an image, an image stabilization module 1 14 as described above 
making it possible to produce stabilized sub-images as an output. These sub-images are sent 
to recording means 116 and, simultaneously, to a display screen such as a liquid crystal 
• screen 118 enabling the operator to display the sequence of recorded images. 



20 



